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VIDEO OVERLAY 



Special effects such as graphic overlaying allow inserting graphics objects, such as borders, titles 
and dates, into the pictures or the real-time video. 

The color-key overlaying method is based on a pre-defined "color-key" value which is used to 
indicate when a graphic pixel is transparent. When it is NOT set to the color key value, it is NOT 
transparent, so it must replace the current pixel of the input picture. Because the overlaying 
needs to be performed very accurately, the switching decision occurs for each pixel of the 
picture, in the 4:4:4 space (3 components per pixel). Therefore the graphic overlay is created in 
the 4:4:4 format (YUV or RGB). Usually, video or pictures are present in the 4:2:0 fonnat in 
systems; therefore, devices need to convert them into tfi^^4T4~space in T order to perform the 
overlaying operation. Reverse conversion is also required since 4:2:0 format is used for the 
encoding. 

- The disclosed device provides a (possibly non real time) pre-processing method to transform 
the graphic overlay into the Y,Cb,Cr 4:2:0 space, with some specific weight calculation 
applied to the chrominance (Cb,Cr) components. To help real time operations, those pre- 
calculated weight values may optionally be embedded or attached to this resulting 4:2:0 
graphic overlay. 

- It provides the associated switching method to be then applied (in real time) to the input 
picture or the video stream directly in 4:2:0 space: Pixel switching on the Luminance (Y) and 
weighted switching on the chrominance (Cb,Cr). 

- The overall system is presented using a simple interpolation method however, other types of 
interpolations methods may also be defined using the same approach. 

- Finally the device can also be extended to alpha blending overlaying. 

- 4:2:0 to 4:4:4 and, 4:4:4 to 4:2:0 real time transforms, as well as YUV to RGB color 
conversions may be removed, with a large computational saving. 

- The luminance switching quality is unchanged since the luminance is not re-sampled. 

- The chrominance switching quality is weighted in order to offer the equivalent quality as 
devices using formats conversions. 



ABSTRACT: 

The purpose of this invention is to provide a simple and efficient way to perform real time color 
key switching directly in the 4:2:0 Y,Cb,Cr space, with a specific associated preprocessing stage 
to transform the usual color key graphic overlay from the 4:4:4 space into the 4:2:0 space. This 
preprocessing stage is not real time constrained since the graphic overlay is selected by the user, 
and can be prepared in a reasonably long period of time. 

Because the overlay switching is done in the 4:2:0 space, a weighted switching is applied to the 
chrominance samples, while usual pixel switching is applied to the luminance samples. 

L Introduction: 

New generation Multimedia wireless products now offer picture and video capture and 
compressions capabilities. Special effects such as graphic overlaying allow inserting graphics 
objects, such as borders, titles and dates, into the pictures or the real-time video. 
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Usually picture or video compression engines use internally the Y,Cb,Cr 4:2:0 format, while the 
overlay color-key switching is defined in the RGB 4:4:4 space, and needs to be operated in this 
space, since it is a pixel switching based method. 

For this reason the incoming video needs to be converted into the 4:4:4 space (optionally by 
applying a YUV to RGB matrix), then the overlaying process is applied. Finally the reverse 
transforms are required, prior to using the encoder for compressing the signal. 

Extra cost brought by these color space and formats conversions have motivated the study of this 
novel method which combines both pixel based switching and weighted (alpha-blending) 
switching. The pixel based switching is applied to the luminance, while the weighted switching 
is applied to the chrominance components. 

Finally a large computation saving is reached since the real time format conversions are removed 
and the switching is applied to a more reduced number of samples defined by the 4:2:0 space. 
For this last reason, the memory requirement is also reduced. 



2. General concept: 



2.L Overlay and color key definition in 4:4:4 space: 

- Let's define the graphic overlay as a picture OVLY444 where a color level (here Y,Cb,Cr) 
4:4:4 is reserved to define if a pixel is transparent or not. This level is called CK (Color Key). 

Example: CK = [0(y), 0(cb), 0(cr)] 

- Let's define the input picture to be PIC444, with the same size in the same 4:4:4 space than the 
OVLY picture. 

We can define the normal overlay process as the following: 

For each PIC444[Y,Cb,Cr] input sample, if the corresponding OVLY444[Y,Cb,Cr] sample has 
the value CK, the picture sample is maintained, otherwise, it is replaced by the 
OVL Y444[ Y,Cb,Cr] sample. 



Algorithm: 




With: j varying from 0 to picture width and i varying from 0 and picture height. 
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Input picture — Graphic overlay. 



X Luminance (Y) 

Chrominance (Cb Cr) 



Figure 1: Pixel switching mechanism 
22. Format representations 

The 4:4:4 to 4:2:0 conversion can be viewed as a linear or bilinear transform that doesn't affect 
the luminance samples, and that divides by two the resolution of the chrominance samples in 
both the horizontal and the vertical directions. 

We represent here the spatial location of the chrominance samples associated to their luminance 
samples in both the 4:4:4 space and the 4:2:0 space: 

- In the 4:4:4 space the chrominance samples apply directly to the same luminance samples 
location. 

- In the 4:2:0 space the chrominance samples apply to surrounding luminance samples and are 
defined in location where contributions to the top and bottom are equal. Note that other type 
of 4:2:0 formats may be used based on other phase location for the chrominance. 
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4:4:4 «:0 

® ® 
® ® 
® ® 
® ® 

O Chrominance (CbCr) X Luminance (Y) 

Figure 2: 4:4:4 and 4:2:0 representations 

2.3. Simplified 4:4:4 to 4:2:0 conversion: 

We define here a simple transform from 4:4:4 to 4:2:0. This transform has been chosen because 
it maintains a very acceptable quality while the complexity is not too high- hi addition the 
chrominance samples only depend on the adjacent surrounding pixels. 

T = (A/8 + B/4 + C/8) + (D/8 + E/4 + F/8) (see fig. 3) 

Where: T is the resulting chrominance sample in the 4:2:0 space 

And: A,B,CJD,E 5 F are surrounding chrominance samples from the 4:4:4 space. 
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@) Chrominance 4:2:0 (Cb Cr) 



T = (A/8+B/4 + <y8)+ (D/8 + E/4+ F/8) 



Figure 3: Chrominance 4:4:4 to 4:2:0 



Note: 



This simplified conversion cannot be applied directly to the color key graphic overlay since 
some overlay pixels may or may not have a valid contribution into the 4:2:0 space. 



The following method is simply based on the use of the 4:4:4 to 4:2:0 chrominance interpolation, 
however, only the non transparent chrominance samples must contribute to the interpolation. 

The process can be separated into the luminance process where the 4:2:0 and 4:4:4 spaces do not 
differ, and the chrominance process where the 4:2:0 and 4:4:4 space are different. 

We define several rules: 

1- OVLY luminance (Y) level 0 is reserved for "transparency", levels [1-255] are the active 
levels containing the luminance value to replace the respective luminance sample in the 
4:2:0 picture. 

2- When a Y pixel is active [1-255] its associated chrominance (Cb,Cr) levels have an active 
contribution level that "replaces", part of the chrominance sample of the 4:2:0 picture, 
proportionally to a "weight*. 

3- When a Y pixel is "transparent 11 the contribution of its associated (Cb,Cr) samples is 
zero. 

4- The total "weighf'of contribution of the (Cb,Cr) 4:4:4 samples into the 4:2:0 (Cb,Cr) 
sample can be defined by the sum of the '"weights" of those of the active contributors. For 
example each individual weight is 1/8 or 1/4 while the total weigh <= 1. (based on T = 
(A/8 + B/4 + C/8) + (D/8 + E/4 + F/8)). 



3. 



Modifying the graphic Overlay into the 4:2:0 space: 
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3. /. Overlay transform (non real time preprocessing): 

Based on the above definitions and rales we can now define the transform of the graphic overlay 
itself: 

- Let's define the 4:2:0 target overlay graphic as OVLY420. 



Luminance: 

Algorithm: 




With: j varying from 0 to picture width and i varying from 0 and picture height. 

For the luminance we simply ensure that when a Y sample is transparent, Y is set to zero,- 
otherwise it must not be a zero value. 

Chrominance: 



Algorithm: 




With: 1 varying from 0 to picture width/2 and k varying from 0 and picture height/2. 
In our example below we obtain: 
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T - 0 A/8 + 1 B/4 + 1 C/8 + 1 D/8 + 0 E/4 + 0 F/8 

0 Yb Yc 

® ® © O O 



® ® ® O O 



Yd 



HP Chrominance 4:2:0 (Cb Cr) 

Luminance: 0 transparent 
Yx active) 

Figure 4: Chrominance switch and weighted contributions 4:4:4 to 4:2:0 
Picture bordering: 

When A an D are beyond the left border position (line beginning) they do not exist, instead we 
can set A = B, and D = E. thus the overall weighting definition is maintained. The same rule 
applies to the associated weight calculation. 

3.2. Associated Weight: 

For each calculated chrominance 4:2:0 samples, from the 4:4:4 overlay graphic, a weight (W) 
can be calculated: This is simply done by adding each individual weight when its respective 
luminance sample is active. This weight value is used at overlaying level to perform an alpha 
blending operation as shown in the overlaying process. 

In our chosen example we obtain: 

W(T) = 01/8 + l 1/4 + 1 1/8 + 1 1/8 + 01/4 + 0 1/8 
W(T) = l/2 

Note: 

The weighting value may be embedded within the chrominance value if the chrominance number 
of levels is limited, such that enough free bits are left available for it. In our example, 4 bits are 
required to store the weight. Otherwise the transparency level can also be associated with each 
chrominance sample, but this requires more data storage. Finally the weight can also be 
reconstructed during the overlay switching operation, at the cost of real time operations. 
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5. 3. 4:2:0 overlaying (real time process): 

Let's define PIC420 the 4:2:0 input picture with the same size and format than the OVLY420 
overlay graphic. 

It is now possible to do the luminance switch, and to do a weighted chrominance switch, since 
each Y from OVLY420 contains either the Y active values or inactive status. Thus the 
chrominance weight reconstruction simply uses the surrounding Y active status, finally Cb,Cr 
contains the interpolated contribution of OVLY444 to be weighted with the Cb,Cr samples of the 
picture. 

Luminance: " ' 



Algorithm: 




The luminance process is trivial since it is just replacing Y samples of the picture when Y 
samples of the overlay are not zero. 



Chrominance: 



Algorithm: 
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Here the largest part of the calculation is to reconstruct the total weight, however, as stated 
earlier, the weighting may be provided attached to (or within) the OVLY420 graphic overlay, 
and may not need to be reconstructed. 

Finally, the total weight allows switching proportionally (weighted) contributions from the 
graphic overlay (W) replacing with the same amount from the input picture sample contribution 

(I-W). 

4:2:0 picture 4:2:0 overlay 



XO XI X2 0 Zl Z2 




23 X4 X5 



4;2:0 result j 

Chrominance (Cb Cr) 
X Z Luminance ( Y) 

Figure 5: Luminance switch / Chrominance weighted switch 
4. Extension to tlie alpha blending overlaying: 

The domain of application of this invention can be extended very directly to alpha blending 
overlaying: 

Using the switch case: 

W(T) = 0,1 /8 + 0,1 14 + 0,1 /8 + 0,1 /8 + 0,1 /4 + 0,1 /8 

We can extend it to: 



Wa(T) 



= aa/8 +ab/4 + ac/8 + ad/8 + ae/4 + af/8 
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•Where o <= a <= 1 represents the opacity value of each luminance pixel of the graphic overlay. 

Ya Yb Yc 

® ® © o o 

(B) (D — 0 0 0 

Yd Ye Yf 



fP Chrominance 4:2:0 (Cb Cr) 

Luminance: 0 transparent 
Yx active) 

Figure 6: Chrominance Alpha weighted contributions 4:4:4 to 4:2:0 

In our example we obtain for T: 

T = A aa/8 + Bab/4 + C ac/8 + D ad/8 + E ae/4 + F af/8 
WafT) = aa/8 + ab/4 + ac/8 +- ad/8 + ae/4 + af/8 

4.L Overlay transform (non real time preprocessing): 

In the same manner the graphic is transformed into 4:2:0 by using individual a values instead of 
Oor 1: 

Luminance: 

For the luminance we pre-multiply the samples by the associated a values. 

Algorithm: 



Chrominance: 
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For the chrominance we interpolate the chrominance sample and the weighting valuerAlgorithm: 




4.2. 4:2:0 overlaying (real time process) : 

The rest of the process uses normal alpha blending for the luminance, as in 4:4:4. 
Luminance: 



Algorithm: 
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Chrominance: 

For the chrominance part it uses also alpha blending (as in the color key overlay), with alpha 
being the Wa value replacing the W weighting value from the color key device. 

Algorithm: 




Note: 

For both the luminance and the chrominance the a and Wa values will be placed into an array 
attached to the 4:2:0 graphic overlay, so that the real time process can access to them. 



This invention offers an original and very efficient way to overlay graphics/colour-key onto 
pictures or real time video flows. For both speed and architectural reasons, this method was 
invented to allow overlay into the YUV 4:2:0 space instead of 4:4:4. 

Implementing color-key based switching overlaying on DSP requires a lot of operations and is 
defined in the RGB 4:4:4 space. The result must then be transformed into the overlaying directly 
into the YUV 4:2:0 space before it is encoded (compressed). The proposed solution allows to do 
the overlaying directly into the YUV 4:2:0 space and saves YUV 4:2:0 to YUV 4:4:4 and then 
YUV 4:4:4 to YUV 4:2:0 transform stages. Thus the largest part of cycles can be saved into 
systems with embedded processors such as DSP. 

Usual known solutions perform the pixel overlaying (switching), into the RGB or YUV 4:4:4 
space. Switching must be done on each component. Then 4:4:4 to 4:2:0 transform is required 
when coming back to the 4:2:0 usual encoding format. 

The proposed solution provides a specific method to preprocess the graphic YUV 4:4:4 color- 
key pictures into a YUV 4:2:0 space (non real time constrained) with some additional 
information embedded called weighting. Then specific 4:2:0 space switching is provided, (real 
time constrained operation). This stage contains minimized calculations or operations. 

The solution is simple in a sense that is saved many MlPS/calculation expensive transform 
stages, without compromising the visual -quality of the result. Also memory buffering 
requirements are reduced, since there is no need to work on the 4:4:4 space (half space required 
in 4:2:0). 
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CLAIMS: 

1 . A preprocessing method for graphic data to transform a graphic overlay into a 
transform space. 

2. the material of this description in any novel or inventive combination. 
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